Measuring  Effects  and  Success  in  Influence 

Operations:  Challenges,  Limitations  and  Opportunities 


Prepared  By: 

Krystel  Carrier-Sabourin 
Royal  Military  College  of  Canada 


SOW  Number:  2009-0302-SLA  -  PR10021 


The  scientific  or  technical  validity  of  this  Contract  Report  is  entirely  the  responsibility  of  the  Contractor  and  the  contents  do  not 
necessarily  have  the  approval  or  endorsement  of  Defence  R&D  Canada. 


Defence  R&D  Canada  -  Toronto 

Contract  Report 

DRDC  Toronto  CR  2011-045 

April  201 1 


Principal  Author 

Original  signed  by  Kiystel  Carrier-Sabourin 
Krystel  Carrier-Sabourin 

War  Studies  doctoral  student,  Royal  Military  College  of  Canada 

Approved  by 

Original  signed  by  Matthew  A.  Lauder  and  Neil  Chuka 
Matthew  A.  Lauder  and  Neil  Chuka 
Contract  Scientific  Authority 

Approved  for  release  by 

Original  signed  by  Joseph  V.  Baranski 
Joseph  V.  Baranski 

Chair,  Knowledge  and  Information  Management  Committee 

Chief  Scientist 


Abstract 


Little  research  in  either  the  military  or  civilian  sphere  has  attempted  to  assess  the  effects, 
degree  of  effects,  and  success  of  influence  operations,  particularly  in  hostile 
environments  where  it  may  prove  both  difficult  and  unreliable  to  conduct  traditional 
interviews  and  opinion  polls.  Yet  it  has  been  asserted  that  the  decisive  effects  of  influence 
operations  are  apparent,  and  can  therefore  be  measured.  This  report  examines  the 
challenges,  limitations,  and  opportunities  inherent  to  the  measurement  of  the  effects  and 
success  of  influence  operations  on  civilian  populations  in  hostile  environments.  The 
report  discusses  the  limitations  of  current  methodologies  and  the  opportunities  associated 
with  unobtrusive  measures.  This  discussion  is  based  on  a  systematic  review  of  the 
literature  in  the  fields  of  psychology,  anthropology  and  sociology,  and  interviews 
conducted  with  pioneers  of  artificial  intelligence  technology  as  a  conflict  early-warning 
system.  Artificial  intelligence  of  social  media  could  be  used  to  assess  the  effectiveness 
and  success  of  influence  operations  in  hostile  environments,  especially  when  attitudes  or 
behaviours  may  be  socially  sensitive  and  might  not  lend  themselves  well  to  participant 
self-assessment,  or  when  pre-effect  data  collection  capacities  are  limited  and  the 
establishment  of  a  baseline  is  not  possible. 
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Resume 


Peu  de  recherches  dans  les  spheres  militaire  ou  civile  ont  ete  menees  pour  tenter 
d’evaluer  les  effets,  le  degre  des  effets  et  le  niveau  de  reussite  des  operations  d’influence, 
en  particular  dans  les  milieux  hostiles  ou  il  pourrait  s’averer  difficile  et  peu  concluant  de 
realiser  des  entrevues  et  des  sondages  d’opinion  par  des  methodes  traditionnelles. 
Cependant,  il  a  ete  affinne  que  les  effets  decisifs  des  operations  d’influence  etaient 
observables  et,  par  le  fait  meme,  mesurables.  Le  present  rapport  etudie  les  defis,  les 
limites  et  les  possibilites  de  1’ evaluation  des  effets  et  du  niveau  de  reussite  des  operations 
d’influence  effectuees  aupres  de  populations  civiles  en  milieux  hostiles.  Il  traite 
egalement  des  limites  des  methodes  utilisees  actuellement,  ainsi  que  des  possibilites  liees 
aux  mesures  discretes.  La  demarche  se  fonde  sur  un  examen  systematique  de  documents 
issus  des  domaines  de  la  psychologie,  de  l’anthropologie  et  de  la  sociologie,  ainsi  que  sur 
des  entrevues  menees  aupres  de  pionniers  des  technologies  de  l’intelligence  artificielle 
utilisees  par  exemple  pour  les  systemes  d’alerte  rapide  lors  de  conflits.  Il  serait  possible 
de  recourir  a  l’intelligence  artificielle  des  rnedias  sociaux  afin  d’evaluer  l’efficacite  et  le 
niveau  de  reussite  des  operations  d’influence  en  milieux  hostiles,  en  particulier  lorsque 
les  attitudes  ou  les  comportements  sont  delicats  sur  le  plan  social  et  ne  se  pretent  pas  bien 
a  1’ auto-evaluation  des  participants,  ou  lorsque  les  capacites  de  collecte  des  donnees 
avant  la  tenue  des  operations  sont  limitees  et  qu’il  est  impossible  d’etablir  une  base  de 
reference. 
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Executive  Summary 


Little  research  in  either  the  military  or  civilian  sphere  has  attempted  to  assess  the  effects,  degree 
of  effects,  and  success  of  influence  operations.  As  a  result,  there  exists  no  agreed-upon  evidence- 
based  methodology  to  measure  the  effects  and  success  of  influence  operations  on  civilian 
populations,  particularly  in  hostile  environments  where  it  may  prove  both  difficult  and  unreliable 
to  conduct  traditional  interviews  and  opinion  polls.  Yet  it  has  been  asserted  that  the  decisive 
effects  of  influence  operations  are  apparent,  and  can  therefore  be  measured. 

This  report  examines  the  challenges,  limitations,  and  opportunities  inherent  to  the  measurement 
of  the  effects  and  success  of  influence  operations  on  civilian  populations  in  hostile  environments. 
Following  a  definition  of  the  effects  and  success  of  influence  operations,  the  report  discusses  the 
limitations  of  current  methodologies,  such  as  the  Tactical  Conflict  Assessment  Framework,  and 
the  opportunities  associated  with  unobtrusive  measures.  This  discussion  is  based  on  a  systematic 
review  of  the  literature  in  the  fields  of  psychology,  anthropology  and  sociology,  and  interviews 
conducted  with  pioneers  of  artificial  intelligence  technology  as  a  conflict  early-warning  system. 

The  report  concludes  that  artificial  intelligence  of  social  media  could  be  used  to  assess  the 
effectiveness  and  success  of  influence  operations  on  civilian  populations  in  hostile  environments, 
especially  when  attitudes  or  behaviours  may  be  socially  sensitive  and  might  not  lend  themselves 
well  to  participant  self-assessment,  or  when  pre-effect  data  collection  capacities  are  limited  and 
the  establishment  of  a  baseline  is  not  possible.  Digital  infonnation  analysis  is  likely  to  gain 
relevance  in  the  measurement  of  the  effects  and  success  of  influence  operations  given  rapid  rates 
of  democratization  of  Internet-based  communications. 


Sommaire 


Mesurer  les  effets  et  la  reussite  des  operations  d'influence 

Peu  de  recherches  dans  les  spheres  militaire  ou  civile  ont  ete  menees  pour  tenter  d’evaluer  les 
effets,  le  degre  des  effets  et  le  niveau  de  reussite  des  operations  d’influence.  C’est  pourquoi  il 
n’existe  pas  de  methodologie  convenue  fondee  sur  des  preuves  pour  mesurer  les  effets  et  la 
reussite  des  operations  d’influence,  en  particulier  dans  les  milieux  hostiles  ou  il  pourrait  s’averer 
difficile  et  peu  concluant  de  realiser  des  entrevues  et  des  sondages  d’opinion  par  des  methodes 
traditionnelles.  Cependant,  il  a  ete  affirme  que  les  effets  decisifs  des  operations  d’influence  sont 
observables  et,  par  le  fait  meme,  mesurables. 

Le  present  rapport  etudie  les  defis,  les  limites  et  les  possibilites  de  1’ evaluation  des  effets  et  du 
niveau  de  reussite  des  operations  d’influence  effectuees  aupres  de  populations  civiles  en  milieux 
hostiles.  En  se  fondant  sur  une  definition  des  effets  et  de  la  reussite  des  operations  d’influence,  le 
rapport  traite  des  limites  des  methodes  utilisees  actuellement,  comine  le  «  Tactical  Conflict 
Assessment  Framework  »,  ainsi  que  des  possibilites  liees  aux  mesures  discretes.  La  demarche  se 
fonde  sur  un  examen  systematique  de  documents  issus  des  domaines  de  la  psychologie,  de 
l’anthropologie  et  de  la  sociologie,  ainsi  que  sur  des  entrevues  menees  aupres  de  pionniers  des 
technologies  de  1’ intelligence  artificielle  utilisees  par  exemple  pour  des  systemes  d’alerte  rapide 
lors  de  conflits. 

Le  rapport  conclut  qu’il  serait  possible  de  recourir  a  1’ intelligence  artificielle  des  medias  sociaux 
afin  d’evaluer  l’efficacite  et  le  niveau  de  reussite  des  operations  d’influence  en  milieux  hostiles, 
en  particulier  lorsque  les  attitudes  ou  les  comportements  sociaux  sont  delicats  sur  le  plan  social 
et  ne  se  pretent  pas  bien  a  1’ auto-evaluation  des  participants,  ou  lorsque  les  capacites  de  collecte 
des  donnees  avant  la  tenue  des  operations  sont  limitees  et  qu’il  est  impossible  d’etablir  une  base 
de  reference.  Il  est  fort  probable  que  le  recours  a  l’analyse  des  informations  numeriques  pour 
mesurer  les  effets  et  la  reussite  des  operations  d’influence  s’intensifie  en  raison  de  la 
democratisation  rapide  des  communications  par  Internet. 
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Introduction:  Current  Assessment  Capabilities 


Several  military  strategists1 2 3  have  argued  that  modern  warfare  has  evolved  from  a 
battle  for  physical  terrain  to  a  battle  for  ‘hearts  and  minds’.  Whereas  the  center  of 
gravity^  in  conventional  warfare  was  seen  as  the  will  of  the  enemy’s  government  and 
military  forces4,  the  centre  of  gravity  in  modern  conflicts  has  shifted  “to  the  perceptions 
of  populations.  Victory  will  be  defined  more  in  terms  of  capturing  the  psycho-cultural 
rather  than  the  geographical  highground”5.  The  infonnation  environment,  and  influence 
operations  more  specifically,  have  therefore  gained  prominence  on  the  modem  battlefield. 

As  a  result  of  this  shift,  measuring  the  success  of  military  campaigns  is  now  an 
inherently  more  difficult  process.  ‘Success’  used  to  be  measured  by  drawing  new  lines  on 
maps  to  demonstrate  newly-acquired  territory  and,  in  the  case  of  conflicts  of  attrition,  by 
citing  asymmetrical  body  counts.  Measuring  ‘hearts  and  minds’  won  in  the  course  of 
influence  operations,  on  the  other  hand,  is  a  multifaceted  and  convoluted  enterprise;  in 
fact,  “useful  measures  of  progress  are  so  elusive  that,  when  asked  [in  2005]  how,  if  he 


1  Boyd,  C.  D.  “Army  10  is  PSYOP:  Influencing  More  with  Less.  Military  Review’.  (Vol.  87,  No.  3,  May- 
June  2007);  Claessen,  E.  “Discouraging  Hearts  and  Minds:  Democracies  and  Insurgencies.”  Military 
Review >.  (Vol.  87,  No.  3,  May-June  2007);  Darley,  W.  “The  Missing  Component  of  U.S.  Strategic 
Communications.”  Joint  Force  Quarterly.  (Vol.  47,  2007);  King,  Sarah  B.  “Military  Social  Influence  in  the 
Global  Information  Environment:  A  Civilian  Primer.”  Analyses  of  Social  Issues  and  Public  Policy.  (Vol. 

00,  No.  00,  2010);  Murphy,  D.  “Attack  or  Defend?  Levering  Information  and  Balancing  Risk  in 
Cyberspace.”  Military  Review.  (May-June  2010);  Scales,  R.  “Clausewitz  and  World  War  IV.”  Armed 
Forces  Journal.  (Vol.  16,  No.  24,  July  2006). 

2  King,  Sarah  B.  “Military  Social  Influence  in  the  Global  Information  Environment.”  p.l. 

3  The  center  of  gravity  in  warfare  is  a  concept  introduced  by  Clausewitz,  which  refers  to  “the  hub  of  all  power 
and  movement,  on  which  everything  depends.”  Clausewitz,  C.  On  War.  in  Howard,  M.  and  Paret,  P.  (eds.  and 
trans.).  (Princeton,  NJ:  Princeton  University  Press,  1976),  p.  485. 

4  Krepinevich,  A.  “Are  We  Winning  In  Iraq?”  (Testimony  to  the  United  States  House  of  Representatives 
Committee  on  Armed  Forces,  Center  for  Strategic  and  Budgetary  Assessments,  March  17,  2005).  p.4. 

5  Scales,  R.  “Clausewitz  and  World  War  IV.”  Armed  Forces  Journal.  (Vol.  16,  No.  24,  July  2006).  p.  1 8. 
http://www.strategicstudiesinstitute.army.mil/Pubs/display.cfm?PubID=642. 
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returned  in  six  months,  he  would  know  whether  progress  was  being  made  [in  the 
counterinsurgency],  General  Fraser  said  he  would  ‘ask  some  Afghans’”6,  as  it  will  be  the 
attitudes,  perspectives,  perceptions,  commitments  and  behaviours  of  the  population  that 
will  ultimately  determine  whether  the  operation  has  been  successful. 

Yet  General  Fraser  did  not  elaborate  on  which  Afghans  to  ask,  what  to  ask,  and 
how  to  ask  it.  Little  research  in  either  the  military  or  civilian  sphere  has  attempted  to 
assess  the  effects,  degree  of  effects,  and  success  of  influence  operations.  As  a  result,  there 
exists  no  agreed-upon  evidence-based  methodology  to  measure  the  effects  and  success  of 
influence  operations,  particularly  in  hostile  environments  where  it  may  prove  both 
difficult  and  unreliable  to  conduct  traditional  interviews  and  opinion  polls.  Looking  at 
American  influence  operations  in  Iraq  as  an  example,  it  is  evident  that  a  consistent 
methodology  is  not  being  applied  to  set  and  analyze  measures  of  effectiveness:  while 
“some  U.S.  units  understand  and  use  the  concepts  of  setting  objectives,  developing 
themes,  and  setting  measures  of  effectiveness...  others  do  not  understand  the  process  and 
therefore  are  just  conducting  operations  without  any  measure  of  success  or  failure.”7 8 9  In 
instances  where  there  are  attempts  at  measuring  the  effects  of  influence  operations,  such 
measurements  are,  at  best,  “difficult  to  ascertain”  given  the  multiple  variables  at  play. 

As  a  result,  “success  in  most  of  the  domains  of  information  operations  is  difficult  to 

6  Graham,  B.  “Enemy  Body  Counts  Revived:  U.S  is  Citing  Tolls  to  Show  Success  in  Iraq.”  Washignton  Post. 
(Washington,  October  24,  2005). 

7  Center  for  Army  Lessons  Learned  (CALL).  “On  Point:  The  United  States  Army  in  Iraqi  Freedom.”  (Fort 
Leavenworth,  August  2004).  p.5. 

8  Lamb,  C.  Review  of  Psychological  Lessons  Learned  from  Recent  Experience.  (Washington,  DC:  National 
Defense  University  Press,  2005).  p.57.  http://www.fas.org/irp/eprint/lamb. pdf 

9  Lamb,  C.  Review  of  Psychological  Lessons  Learned  from  Recent  Experience',  Seese,  G.  And  Smith,  P. 
“Measuring  PSYOP  Effectiveness.”  Special  Warfare.  (Vol.  21,  No. 6,  November-December  2008). 
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measure.”10  A  systematic  ‘culture  of  measurement’  is  also  not  in  place.  During  interviews 

conducted  as  part  of  a  lessons-leamed  exercise  on  psychological  operations  (PSYOP),  it 

was  noted  that  requesting  financial  resources  to  conduct  a  survey  in  a  military 

headquarters  is  “considered  unusual”* 11,  and  that  “PSYOP  forces  lack  the  training  to  do 

polling  systematically  and  professionally  so  that  effects  can  be  more  reliably 
12 

measured”  . 

Despite  these  limitations,  it  has  been  asserted  that  the  decisive  effects  of  influence 
operations  are  apparent  ,  and  can  therefore  be  measured.  These  measures  tend  to  be 
qualitative  in  nature,  and  have  confirmed  “desired  effects.”14  PSYOP  in  Afghanistan,  for 
example,  are  reported  to  have  resulted  in  high  rates  of  Taliban  troop  surrender  during  the 
initial  phases  of  invasion15  and  in  a  high  rate  of  civilian  non-interference  both  during  and 
in  the  immediate  aftermath  of  combat,  thereby  minimizing  civilian  casualties16.  These 
oft-cited  variables,  however,  are  relatively  easy  to  measure  and  do  not  provide  a  complete 
narrative  of  the  effects  and  outcomes  of  influence  operations.  This  is  particularly  the  case 
for  measures  of  effectiveness  of  the  perceptions,  attitudes  and  behaviours  of  civilian 
populations,  since  these  measures  are  “far  more  questionable — and  even  harder  to 


10  Fontenot,  Gregory,  et  al.  “On  Point:  The  United  States  Army  in  Operation  Iraqi  Freedom.”  (First  Naval 
Institute  Press,  2005).  p.419 

11  Lamb,  C.  Review  of  Psychological  Lessons  Learned  from  Recent  Experience,  p.190. 

12  Idem. 

13  Maiers,  Mark  W.  and  Rahn,  Timothy  L.  “Information  Operations  and  Millennium  Challenge.”  Joint 
Force  Quarterly.  (No.  35,  2004).  p.83. 

14  Lamb,  C.  Review  of  Psychological  Lessons  Learned  from  Recent  Experience,  p.27. 

15  Lamb,  C.  Review  of  Psychological  Lessons  Learned  from  Recent  Experience,  p.58. 

16  Dunbar,  R.  “Achieving  Irreversible  Momentum.”  IOSPHERE,  Journal  of  the  Joint  Information 
Operations  Centre.  (Winter  2009);  Lamb,  C.  Review  of  Psychological  Lessons  Learned  from  Recent 
Experience,  p.59. 
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measure”  —  than  measures  of  effectiveness  on  enemy  troops,  which  are  facilitated  by 
the  possibility  of  conducting  interviews  with  prisoners  of  war.  The  indicator  of  civilian 
compliance  during  combat  operations  also  reveals  more  about  the  will  of  the  population 
for  immediate  survival,  rather  than  the  internalization  of  messages  disseminated  over  the 
course  of  influence  operations.  While  the  effects  and  success  of  influence  operations  on 
populations  can  be  measured,  the  selection  of  indicators,  as  well  as  the  collection  and 
analysis  of  data,  so  far  has  been  “inadequate”  . 

This  report  will  inject  social  influence  in  modem  military  affairs  by  looking  at  the 
challenges,  limitations,  and  opportunities  inherent  to  the  measurement  of  the  effects  and 
success  of  influence  operations  on  civilian  populations  in  hostile  environments. 
Following  a  definition  of  the  effects  and  success  of  influence  operations,  this  report  will 
discuss  the  limitations  of  current  methodologies,  and  will  discuss  the  opportunities 
associated  with  unobtrusive  measures. 


Defining  Influence  Operations  Effects  and  Success 

The  ultimate  goal  of  influence  operations  is  to  manoeuvre  a  shift  in  enemy 
decisions  and  population  opinions  in  ways  favourable  to  national  objectives.  A  shift 
occurs  when  one  or  several  effects,  that  is  “the  power  to  bring  about  a  result”19  of  the 


17  King,  Sarah  B.  “Military  Social  Influence  in  the  Global  Information  Environment.”  p.7.  See  also:  Jones, 
J.,  Kuehl,  D.,  Burgess,  D.  and  Rochte,  R.  “Strategic  Communication  and  the  Combatant  Commander.” 
Joint  Force  Quarterly.  (Vol.  55,  No.  4,  2009);  Lamb,  C.  Review  of  Psychological  Lessons  Learned  from 
Recent  Experience',  Shanker,  T.  and  Herding,  M.  “The  Military-media  Relationship:  A  Dysfunctional 
Marriage?”  Military’  Review.  (Vol.  89,  No.  5,  September-October  2009). 

18  Lamb,  C.  Review  of  Psychological  Lessons  Learned  from  Recent  Experience,  p.6. 

19  Mann,  Edward  C.,  Endersby,  Gary,  and  Searle,  Thomas  R.  “Thinking  Effects:  Effects-based 
Methodology  for  Joint  Operations.”  CADRE  Paper  No.15.  (Maxwell  Air  Force  Base,  AL:  Air  University 
Press,  October  2002).  p.  30. 
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persuasion,  are  produced  using  either  kinetic  or  non-kinetic  actions,  or  a  combination  of 
both.  For  an  effect  to  occur  there  must  be  a  change  in  the  population’s  decision  cycle  or 
the  manifestation  of  a  differing  behaviour.  In  other  words,  this  shift  must  lead  to 
measurable  effects  that  are  either  attitudinal  (i.e.  awareness,  perspective,  perception, 
commitment)  and/or  behavioural  in  nature.  Measures  of  effectiveness  of  influence 
operations  seek  to  qualify  and  quantify  if  a  message  intended  to  influence  was  perceived 
and  then  evaluated  by  the  target  audience,  if  it  affected  the  target’s  thinking,  and  if  it  led 
to  a  behavioural  consequence.  Examples  of  effects  include,  but  are  not  limited  to, 

information  denial  and  delay,  psycho-social  attitude  shift,  prolonged  decision-making, 

20 

and  ultimately,  change  in  target  audience  behaviour.' 

There  are  different  levels  and  degrees  of  effects.  Levels  of  effects  are  classified 
along  first,  second  and  third  order  effects: 

[F]irst-order  effects  are  associated  with  the  physical  dimension  of  the  information 
environment,  while  second-  and  third-order  effects  are  associated  with  the  information 
environment's  information  and  cognitive  dimensions.  A  first-order  effect  is  a  direct 
effect,  a  result  of  actions  with  no  intervening  effect  or  mechanism  between  the  act  and 
outcome.  Such  an  effect  can  trigger  additional  outcomes,  which  are  indirect  (second- 
and  third-order)  effects.21 

Measures  of  effectiveness  must  therefore  gauge  first,  second  and  third  order  effects,  as 
unanticipated  reactions  to  influence  operations  could  arise  from  first  order  effects,  and 
adversely  impact  the  population’s  attitude  and  behaviour.  Degrees  of  social  influence 
effects  are  situated  on  a  continuum,  ranging  from  conformity,  compliance,  and  obedience 
to  commitment,  and  can  either  be  positive  (i.e.,  confonnity)  or  negative  (i.e.,  non- 

20  Staker,  R.J.  “Military  Information  Operations  Analysis  Using  Influence  Diagrams  and  Coloured  Petri 
Nets.”(DSTO  Electronics  and  Surveillance  Research  Laboratory,  Department  of  Defense,  Salisbury, 
Australia,  1999).  p.iii.  http://dspace.dsto.defence.gov.aU/dspace/bitstream/1947/4354/l/DSTO-TR-0914.pdf 

21  Emery,  Norman  E.  “Understanding  the  Role  of  People,  Capabilities,  and  Effects.”  Military  Review. 
(November-December  2008).  p.27. 
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conformity)  (see  Figure  1).  Measures  of  effectiveness  should,  therefore,  have  a  level  of 
sophistication  that  enables  an  assessment  of  the  degree  and  direction  of  the  effect. 

Figure  1:  Continuum  of  Degrees  and  Directions  of  Social  Effects 


Measuring  effects  and  measuring  success  are  two  different  processes.  Success  occurs 
when  effects  align  with  the  commander’s  objectives.  An  effect,  on  the  other  hand,  can  be 
in  support  of,  or  detrimental  to,  the  commander’s  objectives.  An  effect  could  also  have 
been  induced,  but  not  to  a  sufficient  degree  to  meet  the  commander’s  objectives. 
Measuring  success,  therefore,  involves  measuring  the  sum  of  effects  -  including  first, 
second  and  third  order  effects,  as  well  as  the  degree  and  direction  of  effects  -  to 
determine  whether  the  message  was  perceived,  assessed,  and  assimilated  by  the  target 
audience  so  as  to  meet  the  commander’s  objectives.  The  success  of  influence  operations 
in  irregular  warfare  can  also  only  be  evaluated  over  a  period  of  time,  especially  in  view  of 
the  fact  that  insurgencies  historically  tend  to  between  9  to  12  years'  .  Indicators  of 
success,  therefore,  ought  to  focus  on  trends  in  the  population’s  attitudes  towards  and 

22  Emery,  Norman  E.  “Understanding  the  Role  of  People,  Capabilities,  and  Effects.”  p.  31. 
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23 

behaviour  during  the  war,  as  opposed  to  snapshots  at  a  particular  moment-  —  these 
‘snapshots’,  however,  are  appropriate  for  measuring  effects.  The  commander’s  objectives 
and  target  audiences  must  also  be  well-defined  to  allow  for  a  proper  assessment  of 
success;  “without  stepping  back  to  examine  the  fundamental  premises  of  what  success 
means  in  irregular  warfare,  decision  makers  cannot  have  confidence  in  their 
methodologies  or  results”24.  These  objectives  should,  therefore,  be  specific,  measurable, 
attainable,  realistic,  timely  and  tangible,  by  specifying  the  target  audience,  what  is  to  be 
accomplished,  for  what  purpose,  in  which  location  and  within  which  timeframe,  and  by 
being  associated  with  a  set  of  criteria  that  will  be  used  to  assess  progress  and  recognize 
when  the  overarching  goal  has  been  attained. 


Measures:  Why  Polls  and  Surveys  Alone  Are  Insufficient 

Qualitative  methods  that  have  been  employed  to  assess  the  effects  and  success  of 
information  operations  include  direct  observation,  polling,  surveys,  and  interviews  with  a 

25 

sample  of  the  population"  : 

[The  information  is  collected]  in  face-to-face  meetings  that  take  place  during  patrols; 
focus  groups  with  locals  to  clarify  their  views  and  formulate  an  understanding  of  the 
underlying  structure  of  their  attitudes,  policy  preferences,  and  behaviors;  and  survey 
research  that  can  facilitate  the  creation  of  quantitative  baselines  and  trend  analyses  of 
key  attitudes  or  that  can  predict  attitude  change  based  on  a  knowledge  of  underlying 


23  Krepinevich,  A.,  “Are  We  Winning  In  Iraq?”,  Testimony  to  the  United  States  House  of  Representatives 
Committee  on  Armed  Forces,  Center  for  Strategic  and  Budgetary  Assessments,  March  17,  2005,  p.3. 

24  Clancy,  James  and  Cresset,  Chuck.  “Measuring  Effectiveness  in  IrregularWarfare.”  Parameters.  (Vol. 

37,  2007).  p.99. 

25  Lamb,  C.  Review  of  Psychological  Lessons  Learned  from  Recent  Experience,  p.29. 
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attitude  structures  and,  thereby,  inform  the  development  of  appropriately  targeted  and 
tailored  influence  campaigns  and  messages.26 

These  methods,  particularly  attitude  surveys,  are  seen  as  an  “efficient  way”'  to  assess 
the  effects  and  success  of  influence  messages  on  target  audiences.  As  such,  the 
development,  administration  and  analysis  of  survey  instruments  are  seen  as  key  in  the 
measurement  of  effectiveness  and  success  of  influence  operations.  When  analysts 
allude  to  or  identify  limitations  in  these  data  collection  methods,  they  tend  to  concentrate 
on  the  physical  peril  inherent  to  conducting  surveys  in  dangerous  environments,  and 
emphasize  that  “there  appear  to  be  few  alternatives  [to  surveys  and  polling]  for 
developing  the  sort  of  detailed  understanding  of  attitudes  that  is  necessary  for  effective 
influence  operations”30. 

I  hypothesize,  however,  that  these  data  collection  methods  do  not  generate  accurate  or 
reliable  measures  of  effectiveness  or  success  in  hostile  environments.  First,  analysts  are 
reported  to  be  “overcome  by  the  sheer  volume  of  raw  data”  collected  in  the  context  of 
recent  operations;  as  a  result,  “there  is  little  foundational  understanding  of  what  success 
means  in  irregular  warfare  that  will  assist  analysts  in  interpreting  operational 
effectiveness.”  '  In  the  context  of  influence  operations,  the  absence  of  an  accepted 
analytical  framework  often  renders  the  captured  data  meaningless.  Second,  these  methods 
center  on  participant  self-assessment  that  usually  requires  the  data  collector  (survey 

26Larson,  Eric  V.  et  al.  Foundations  of  Effective  Influence  Operations:  A  Framework  for  Enhancing  Army 
Capabilities.  (RAND  Corporation:  Santa  Monica,  2006).  p.67. 

27  Ibid.  p.70. 

28  Ibid  p.77-78. 

29Ibid.  p.67. 

30ldem. 

31  Clancy,  James  and  Crosset,  Chuck.  “Measuring  Effectiveness  in  IrregularWarfare.”  p.88. 

32  Idem. 


administrator)  be  physically  present;  and,  as  such,  introduce  “a  foreign  element  into  the 
social  setting  they  would  describe.”  The  presence  of  a  civil-military  relations  officer  on 
patrol  discussing  with  village  elders  or  an  indigenous  civilian  researcher  hired  to  conduct 
a  survey  can  be  resented  by  respondents,  who  can  be  offended  by  the  questions  asked  or 
irritated  at  being  asked  to  participate  in  a  survey.  Third,  the  physical  presence  of  the 
surveyor  also  interrupts  the  natural  stream  of  behaviour  of  the  respondents;  and,  as  such, 
not  only  measures  but  also  creates  attitudes  (second  and  third  order  effects).  Classical 
experiments  by  Schanck  and  Goodman  conducted  in  193934,  and  by  Crespi  in  194  8  35, 
demonstrate  that  a  survey  “forces  upon  the  subject  a  role-defining  decision”  ,  and  may 
compel  the  respondent  to  formulate  an  opinion  or  attitude  when  one  might  not  have 
existed  before.  Fourth,  and  most  importantly,  these  data  collection  methods  assess 
attitudes  and  behaviours  that  respondents  may  not  be  willing  to  report  directly,  especially 
in  hostile  environments  where  expressing  support  for  or  opposition  to  a  leader,  a 
government,  an  ideology,  a  military  operation  or  an  anned  force  could  have  life  or  death 
consequences  for  the  respondents.  For  example,  participants  in  a  survey  conducted  by  the 
Canadian  Forces  with  residents  of  Kandahar  in  2010  reported  feeling  that  the  security 
situation  was  improving  -  yet  the  International  Committee  of  the  Red  Cross  reported 


33  Webb,  Eugene  J.  et  al.  Unobtrusive  Measures:  Revised  Edition.  (Sage  Publications:  Thousand  Oaks, 
2000).  p.l. 

34  Schanck,  R.  L.,  and  Goodman,  C.  “Reactions  to  Propaganda  on  Both  Sides  of  a  Controversial  Issue.” 
Public  Opinion  Quarterly.  (Vol.  3,  1939). 

35  Crespi,  Leo  P.  “The  Interview  Effect  in  Polling.”  Public  Opinion  Quarterly.  (Vol.  12,  No.  1,  1948). 

36  Webb,  Eugene  J.  et  al.  Unobtrusive  Measures:  Revised  Edition.  (Sage  Publications:  Thousand  Oaks, 
2000).  p.l. 

37  See  also:  Mercatoris,  M.  and  and  Craighead,  W.  E.  “Effects  of  non-participant  Observation  on  Teacher 
and  Pupil  Classroom  Behavior.”  Journal  of  Educational  Psychology’.  (Vol.  66,  1974);  Surratt,  P.  R.,  Ulrich, 
R.  E.,  and  Hawkins,  R.  O.  “An  Elementary  Student  as  a  Behavioral  Engineer.”  Journal  of  Applied  Behavior 
Analysis.  (Vol.  2,  1969);  White,  G.  D.  “The  Effects  of  Observer  Presence  on  the  Activity  Level  of 
Families.”  Journal  of  Applied  Behavior  Analysis.  (Vol.  10,  1977). 
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(over  the  same  time  period)  the  number  of  civilian  war  casualties  in  the  region  was 

TO 

drastically  increasing  ,  internal  United  Nations  residual  risk  accessibility  maps  showed  a 
marked  deterioration  of  the  security  situation  over  the  same  period,  and  a  survey  released 
by  the  Asia  Foundation  reported  that  61%  of  Afghans  in  the  South  East  feared  for  their 
personal  safety  .  “In  the  context  of  a  military  campaign,  there  are  many  challenges  to 
ensuring  that  the  sorts  of  attitudes  that  are  presented  in  focus  groups  and  surveys  are 
sincere  ones”40.  Surveying  a  population  in  a  hostile  environment  on  socially  sensitive 
issues  may  not  yield  an  accurate  attitudinal  measure. 

Another  example  of  the  fallibility  of  surveys  as  a  data  collection  method  to  measure 
effectiveness  is  the  Tactical  Conflict  Assessment  Framework  (TCAF)  developed  by 
USAID  and  used  by  the  British  52  Brigade  in  Helmand  Province.  The  TCAPF 
questionnaire  was  used  at  the  tactical  level  to  establish  a  baseline  and  measure  the  impact 
of  activities  on  changes  in  local  perceptions  of  the  causes  of  instability.  Using  a 
diagnostic  methodology,  patrols  asked  locals  a  set  of  four  questions  to  identify  what 
issues  most  concerned  them41;  follow-up  questionnaires  were  then  administered  to 
measure  the  effectiveness  of  these  activities.  While  “a  trial  of  TCAF  in  Lashkar  Gah 
appeared  to  show  very  positive  results  in  terms  of  gathering  intelligence  to  target  non- 


3S  What’s  more,  this  number  is  likely  to  be  under-representative  given  that  the  survey  team  did  not  travel  to 
a  number  of  sampling  points  that  were  inaccessible  due  to  security  reasons.  “Kandahar,  Afghanistan, 
Security  ‘Deteriorating’:  ICRC.”  (United  Nations  Radio  transcript,  December  10,  2010). 
http  ://www.  unmultimedia.org/radio/ english/detail/ 1 03  5  85  .html 

39  Tariq,  Mohammad  Osman,  Ayoubi,  Najla,  and  Haqbeen,  Fazel  Rabi.  Afghanistan  in  2010:  A  Survey  of 
the  Afghan  People.  (Asia  Foundation,  November  2010).  p.4. 

40  Mann,  Edward  C.,  Endersby,  Gary,  and  Searle,  Thomas  R.  “Thinking  Effects.”  p.66. 

41  These  questions  are:  Q1  -  Flave  there  been  changes  in  the  village  population  and  why?  Q2  -  What  are  the 
most  important  problems  facing  the  village?  -  Q3  -  Who  do  you  believe  can  solve  your  problems?  Q4  - 
What  should  be  done  first?  Farrell,  Theo.  “Improving  in  War:  Military  Adaptation  and  the  British  in 
Helmand  Province,  Afghanistan,  2006-2009.”  Journal  of  Strategic  Studies.  (Vol.  33,  No.  4,  2010).  p.579. 
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kinetic  activities...  this  may  have  created  unrealistic  expectations  in  the  task  force 

headquarters,  and  it  would  appear  that  TCAF  was  rolled  out  too  quickly  across  Helmand, 

and  especially  into  districts  where  the  environment  was  not  conducive  to  regular 

interaction  with  non-hostile  local  populations”42.  Other  methodological  issues 

surrounding  the  implementation  of  TCAF  include  the  reliance  on  convenience  sampling, 

significant  amounts  of  variation  in  the  selection  of  participants,  the  instructions  given  to 

soldiers  on  how  to  carry-out  interviews,  and  in  the  conduct  of  interviews.43  The  high- 

profile  nature  of  the  interviews,  which  took  place  in  public  forums,  also  had  the  potential 

to  add  response  bias,  particularly  given  that 

[I]t  is  often  said  that  in  Afghanistan  people  will  tend  to  tell  you  what  they  think  you  want 
to  hear,  rather  than  voicing  their  true  opinion  on  a  matter.  This  tendency  might  be 
exacerbated  when...  a  question  such  as  'Who  do  you  believe  can  solve  your  problems?'  is 
asked  by  an  armed  ISAF  soldier;  the  respondent  might  feel  an  even  greater  pressure  to 
respond  in  a  socially  desirable  manner;  thus  the  answers  of  'government'  or  'ISAF'  might 
be  given  even  though  it  may  not  be  their  true  opinion44. 

More  importantly,  the  survey,  developed  as  a  tool  to  measure  the  impact  of  development 
programming  in  conflict-affected  environments,  serves  as  a  better  measure  of  stability 
operations  than  of  influence  operations.45  As  a  result,  the  original  TCAF  was 
discontinued  as  a  survey  instrument  by  the  British  military  in  2009. 


42  Farrell,  Theo.  “Improving  in  War:  Military  Adaptation  and  the  British  in  Helmand  Province, 
Afghanistan,  2006-2009.”  Journal  of  Strategic  Studies.  (Vol.  33,  No.  4,  2010).  p.580. 

43  Wilson,  David  and  Conway,  Gareth  E.  “The  Tactical  Conflict  Assessment  Framework:  A  Short-lived 
Panacea.”  The  Rusi  Journal.  (Vol.  154,  No.  1,  February  2009).  p.10. 

44  Wilson,  David  and  Conway,  Gareth  E.  “The  Tactical  Conflict  Assessment  Framework:  A  Short-lived 
Panacea.”  The  Rusi  Journal.  (Vol.  154,  No.  1,  February  2009).  p.10. 

45  Ibid.  p.  13. 
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Unobtrusive  Measures 


Empirical  evidence  from  the  field  of  social  science  indicates  that  unobtrusive 
measures  “produce  more  valid  attitudinal  estimates  for  issues  that  are  socially  sensitive  in 
nature”46  than  participant  self-assessment.  Unobtrusive  measures  were  first  developed  by 
Webb,  Campbell,  Schwartz  and  Sechrest  in  1966  and  involve  methodologies  that  do  not 
require  direct  elicitation  of  data  from  the  research  subjects.  These  measures  are  described 
as  “nonreactive  methods  of  gathering  data”47  that  do  not  call  for  the  researcher  to  intrude 
in  the  research  context;  as  such,  respondents  are  neither  aware  of  the  existence  of  the 
research,  its  purpose,  nor  that  they  are  being  studied.  These  measures  overcome  a  variety 
of  pressures  likely  to  distort  responses  in  direct  surveys  and  self-report  procedures  for 
assessing  attitudes,  namely:  “evaluation  apprehension,  demand  characteristics,  social 

48 

approval,  the  ‘guinea  pig’  effect,  response  sets,  and  investigator  characteristics  effects.” 
Unobtrusive  measures  have  been  shown  to  reduce  reactivity,  enhance  reliability  of 
results,  and,  when  relying  on  archival  or  trace  evidence,  allow  for  study  over  an  extended 
period  of  time. 

Unobtrusive  measures  can  rely  on  simple  observation,  disguised  observation  (also 
known  as  indirect  measure),  erosion  measures  (signs  and  degree  of  use  yields  the 
measure)  accretion  measures  (research  evidence  is  the  deposit  of  materials),  as  well  as 

46  Divdio,  John  F.  and  Fazio,  Russell  H.  “New  Technologies  for  the  Direct  and  Indirect  Assessment  of 
Attitudes.”  In  Questions  About  Questions:  Inquiries  into  the  Cognitive  Bases  of  Surveys.  Judith  M.  Tanur 
(ed.).  (Russell  Sage  Foundation:  New  York,  1994).  p.228. 

47  Flagan,  Frank  E.  Essentials  of  Research  Methods  in  Criminal  Justice  and  Criminology.  (Prentice  Hall, 
2003).  p.234. 

48Cialdini,  Robert  B.  and  Baumann,  Donald  J.  “Littering:  A  New  Unobtrusive  Measure  of  Attitude.”  Social 
Psychology’  Quarterly.  (Vol.  44,  No.  3,  1981).  p.254. 
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content-analysis  of  archives,  speeches,  mass-media.  In  certain  instances,  more  unusual 
data  sources  have  been  used  for  content-analysis,  such  as  graffiti,  litter  and  obituaries. 
The  greatest  impediments  to  the  use  of  unobtrusive  data  are  ethical  issues,  namely 
concerns  over  the  privacy  and  non-consent  of  research  subjects,  as  well  as  the  difficulty 
of  obtaining  non- archival  unobtrusive  data.49 


Applying  Unobtrusive  Measures  to  Hostile 
Environments 


Yet  new  technology  could  enable  the  application  of  unobtrusive  methods  to  measure 
the  effects  of  influence  operations  in  hostile  environments.  Increased  sophistication  in 
artificial  intelligence,  and  the  democratization  of  digital  media,  are  ushering  in  a  new  era 
for  researchers  through  the  rapid  increase  of  digital  unobtrusive  data  sources.  Online 
communications  create  ‘digital  footprints’,  which  constitute  unobtrusive  data  that  is 
already  transcribed  and  can  be  easily  accessed  and  analysed.50  These  footprints  are  being 
used  in  the  private  sector  by  marketers  to  analyze  attitudinal  and  behavioural  trends 
through  a  content-analysis  of  the  substance,  process,  and  structure  of  online 
communication  networks.  Artificial  intelligence  is  capable  of  ‘reading’  all  forms  of 
digital  text  media  publicly  available  on  the  Internet,  which  allows  for  the  monitoring  and 
analysis  of  the  content  of  social  media  networks,  such  as  Facebook,  Twitter,  and  Bebo, 


49  See:  Kazdin,  Alan  E.  “Unobtrusive  Measures  in  Behavioral  Assessment.”  Applied  Behavioral  Analysis. 
(Vol.  12,  No.  4,  Winter  1979).  p.721;  Sechrest,  Lee  and  Phillips,  Melinda.  “Unobtrusive  Measures:  An 
Overview.”  In  Unobtrusive  Measurement  Today.  Lee  Sechrest  (ed.).  (Jossey-Bass  Inc.  Publishers:  San 
Francisco,  1979).  p.13. 

50  Arosio,  Laura.  “Personal  Documents  on  the  Internet:  What’s  New  and  What’s  Old.”  Journal  of 
Comparative  Research  in  Anthropology  and  Sociology.  (Vol.  1,  No.  2,  2010).  p.35. 
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newsgroups,  online  forums,  weblogs,  micro-blogs,  and  other  Web  2.0  social  media 
services  around  the  world.  Keywords,  phrases,  and  strings  are  analyzed  by  the  artificial 
intelligence  system,  which  organizes  and  categorizes  the  data  according  to  perceptions, 
attitudes,  and  behaviours,  as  well  as  an  aggregate  profile  of  the  research  subjects,  such  as 
gender,  age,  ethnicity,  geographic  location,  and  political  affiliation.  The  attitudinal  and 
behavioural  measures  generated  from  digital  samples  are  representative  of  the  larger 
population  within  a  geographic  area.  In  a  study  published  in  1997  (when  Internet 
penetration  rates  were  relatively  low),  Smith  and  Leigh  compared  “Internet  and  non- 
Internet  samples  on  several  demographic  variables.  They  found  that  the  samples  did  not 
differ  in  terms  of  sexual  orientation,  marital  status,  ethnicity,  education,  and  religiosity, 
but  that  they  did  differ  in  terms  of  age  and  sex.  While  sex  and  age  are  important 
variables,  if  anything  these  results  appear  to  support  the  advantage  of  Internet  over 
traditional  samples  since  the  Internet  sample  contained  a  wider  age  range”51  and  a  greater 
number  of  research  subjects. 

The  categorization  of  data  according  to  perceptions,  attitudes,  and  behaviours,  as  well 
as  an  aggregate  profile  of  the  research  subjects,  allows  analysts  to  measure  change  in 
attitudinal  and  behavioural  variables,  and  to  compare  these  changes  amongst  various 
sophisticated  subject  profiles  (i.e.,  males  in  x  age  category,  living  in  y  area,  with  z  level 
of  education).  Such  content-analysis  could  also  enable  analysts  to  measure  whether 
changes  in  attitudes  and  behaviours  are  a  result  of  nonnative  or  informational  influence. 
“The  first,  nonnative  influence,  is  based  on  the  desire  to  conform  to  the  expectations  of 

51  Hewson,  Claire  et  al.  Internet  Research  Methods  :  A  Practical  Guide  for  the  Social  and  Behavioural 
Sciences.  (SAGE  Publications:  London,  2003).  p.27. 
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others.  Judgment  shifts  are  assumed  to  result  from  exposure  to  others'  choice  preferences 
and  from  subsequent  conformity  to  the  norms  that  are  implicit  or  explicit  in  these 
preferences.  The  second,  informational  influence,  is  based  on  the  acceptance  of 
information  from  others  as  evidence  about  reality.”  Studies  have  shown  that 
informational  influence  produces  more  frequent  and  stronger  shifts  than  does  nonnative 
influence;  analysts  could,  therefore,  have  an  indication  of  the  strength  of  the  attitudinal  or 

ST 

behavioural  shift.  Such  information  would  enable  analysts  tasked  with  measuring  the 
effects  of  influence  operation  to  assess  the  continuum  of  degree  of  effects,  establishing 
whether  behaviours  and  attitudes  are  caused  by  conformity,  compliance,  obedience,  or 
commitment. 

The  greatest  potential  of  digital  information  -  apart  from  the  unobtrusiveness  of  the 
data  collection  method  and  the  large  number  of  research  subjects  -  lies  in  its  archival 
capabilities;  data  tends  to  be  stored  on  servers  for  decades,  allowing  for  the  collection  and 
analysis  of  data  produced  in  the  past.  The  ease  of  access  to  this  archived  information 
enables  analysts  to  establish  baselines  and  create  trend  analyses.  This  capability  is 
particularly  pertinent  to  information  operations  in  hostile  environments  where  it  is 
difficult  to  ascertain  a  baseline  if  pre-effect  data  collection  capacities  are  limited. 

Digital  information  analysis  is  likely  to  gain  relevance  in  the  measurement  of  the 
effects  and  success  of  influence  operations  given  rapid  rates  of  democratization  of 

52  Kaplan,  Martin  F.  and  Miller,  Charles  E.  “Group  Decision  Making  and  Normative  Versus  Informational 
Influence:  Effects  of  Type  of  Issue  and  Assigned  Decision  Rule.”  Journal  of  Personality  and  Social 
Psychology’.  (Vol.  53,  No.  2,  1987).  p.306. 

53  Burnstein,  E.,  &  Santis,  K.  “Attitude  Polarization  in  Groups.”  In  R.  E.  Petty,  T.  M.  Ostrom  and  T.  C. 
Brock  (Eds.).  Cognitive  Responses  in  Persuasion  (Hillsdale,  NJ:  Erlbaum,  1981);  Kaplan,  M.  E,  &  Miller, 
C.  E.  “Group  Discussion  and  Judgment.”  In  P.  Paulus  (Ed.).  Basic  Group  Processes.  (New  York:  Springer- 
Verlag,  1983). 
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Internet-based  communications.  “Technology  use  rates  are  considerably  higher  now  than 
they  were  just  a  few  years  ago  for  almost  every  country  in  the  world”54,  including  failed 
and  fragile  states.  The  number  of  Internet  users  in  the  world  doubled  between  2005  and 
2010.  In  2010,  the  number  of  Internet  users  surpassed  the  2  billion  mark  -  1.6  billion  of 
them  even  had  access  to  the  Internet  at  home  in  2010  (up  from  1.4  billion  in  2009), 
including  1.2  billion  in  developing  countries.55  Indeed,  while  in  2005  the  majority  of 
Internet  users  were  in  the  developed  world,  by  2010  developing  countries  increasing  their 
share  of  Internet  subscriptions  to  57  percent56  of  the  world’s  connections  (see  Graph  1). 
To  put  these  numbers  in  perspective,  while  the  International  Telecommunications 
Union57  reports  that  only  3.4%  of  the  population  of  Afghanistan  had  access  to  the  Internet 
in  2010,  Internet  subscriber  penetration  rates  are  high,  with  33  percent  increase  annually 
-  and  these  rates  are  expected  to  continue  to  increase  exponentially  given  new 
investments  in  Afghan  telecommunications  infrastructure.  Other  countries  of  interest  also 
have  high  Internet  penetration  rate:  Iran  at  34.9  percent,  Haiti  at  11.3  percent,  Sudan  at 
10.0  percent,  and  Lybia  at  5.5  percent.  Examples  from  the  realm  of  development 
indicate  an  appetite  on  the  part  of  villagers  in  the  developing  world  to  use  the  Internet59. 


54  Chinn,  Menzie  D.  and  Fairlie,  Robert  W.  “ICT  Use  in  the  Developing  World:  An  Analysis  of  Differences 
in  Computer  and  Internet  Penetration.”  Review  of  International  Economics.  (Vol.  18,  No.  1,  June  2006). 

p.  155. 

55  “ICT  Facts  and  Figures:  The  World  in  2010  -  The  Rise  of  3G.”  (International  Telecommunication  Union, 
Geneva,  2010).  p.5.  http://www.itu.int/ITU-D/ict/material/FactsFigures2010.pdf 

56  Ibid.  p.4. 

57  The  International  Telecommunications  Union  (ITU)  is  the  leading  United  Nations  agency  for  information 
and  communication  technology  issues,  and  the  global  focal  point  for  governments  and  the  private  sector  in 
developing  networks  and  services. 

58 ITU 

59  Tan-Torres  Edejer,  Tessa.  “Disseminating  Health  Information  in  Developing  Countries:  The  Role  of  the 
Internet.”  British  Medical  Journal.  (Vol.  321,  No.  797,  September  2000). 
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The  high  penetration  rates  of  satellite  television  and  mobile  telephones60  in  the 
developing  world  also  demonstrate  that  new  technologies  are  being  adopted  and 
increasingly  used;  while  accessing  the  Internet  requires  a  greater  capital  investment,  the 
exponential  increase  in  Internet  penetration  rates  suggests  that  unobtrusive  measures 
collected  through  artificial  intelligence  could  be  increasingly  used  to  measures  the  effects 
of  influence  operations  in  hostile  environments. 


Europe  The  Americas  CIS**  World  Arab  States 

*  Estimate 

**  Commonwealth  of  Independent  States 

Regions  are  based  on  the  ITU  BDT  Regions,  see:  http://www.itu.int/ITU- 
D/ict/defmitions/regions/index.html 

Source:  ITU  World  Telecommunication  /ICT  Indicators  database 


Asia  & 
Pacific 


Africa 


Conclusion 

This  report  injected  social  influence  in  modern  military  affairs  by  looking  at  the 
challenges,  limitations,  and  opportunities  inherent  to  the  measurement  of  the  effects  and 

60  Ghashghai,  Elham  and  Lewis,  Rosalind.  “Issues  Affecting  Internet  Use  in  Aghanistan  and  Developing 
Countries  in  the  Middle  East.”  Rand  Issue  Paper.  (2002). 
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success  of  influence  operations  on  civilian  populations  in  hostile  environments. 
Measuring  the  success  of  influence  operations  in  unconventional  warfare,  where  the 
center  of  gravity  is  the  population,  as  opposed  to  governments  or  the  enemy’s  military 
forces,  in  an  intricate  matter.  Whereas  during  the  Cold  War  analysts  measure  the  success 
of  influence  operations  by  looking  at  policy  decisions,61  analysts  must  now  analyse  data 
at  a  micro  level. 

Powerful  artificial  intelligence  tools  are  being  used  by  various  actors,  such  as  the 
United  Nations,  the  private  sector,  governments,  and  academia  to  collect  and  analyse  data 
related  to  complex  policy  issues.  In  fact,  the  technology  is  currently  being  assessed  for 
imminent  use  as  a  conflict  early-warning  system.  As  recent  natural  disasters  and  man¬ 
made  crises  around  the  world  have  shown,  social  and  digital  media  are  increasingly  being 
used  during  emergencies  to  share  news,  coordinate  responses,  express  opinions,  and 
organize  future  action.  It  is  important  to  note,  however,  that  unobtrusive  measures  are 
stronger  and  most  useful  when  used  in  conjunction  with  obtrusive  methods  -  in  the  case 
of  influence  operations,  content-analysis  of  social  media  through  artificial  intelligence 
could  be  supplemented  by  direct  observations  and  interviews  conducted  over  the  course 
of  regular  civil-military  operations.  Although  further  research  is  warranted,  the  review  of 
the  literature  as  well  as  interviews  conducted  with  pioneers  of  this  technology  suggest 
that  artificial  intelligence  of  digital  media  could  be  used  to  assess  the  effectiveness  and 
success  of  influence  operations  in  hostile  enviromnents,  especially  when  such  attitudes  or 
behaviours  may  be  socially  sensitive  and  might  not  lend  themselves  well  to  participant 

61  Yost,  David.  “The  Campaign  Against  INF  in  West  Germany.”  In  Brian  D.  Dailey  and  Patrick  J.  Parker 
(eds.).  Soviet  Strategic  Deception.  (Lexington,  MA:  Lexington  Books,  1987). 
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self-assessment,  or  when  pre-effect  data  collection  capacities  are  limited  and  the 
establishment  of  a  baseline  is  not  possible. 
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discussion  is  based  on  a  systematic  review  of  the  literature  in  the  fields  of  psychology,  anthropology  and 
sociology,  and  interviews  conducted  with  pioneers  of  artificial  intelligence  technology  as  a  conflict  early- 
warning  system.  Artificial  intelligence  of  social  media  could  be  used  to  assess  the  effectiveness  and 
success  of  influence  operations  in  hostile  environments,  especially  when  attitudes  or  behaviours  may  be 
socially  sensitive  and  might  not  lend  themselves  well  to  participant  self-assessment,  or  when  pre-effect 
data  collection  capacities  are  limited  and  the  establishment  of  a  baseline  is  not  possible. 

Peu  de  recherches  dans  les  spheres  militaire  ou  civile  ont  ete  menees  pour  tenter  d’evaluer  les  effets,  le 
degre  des  effets  et  le  niveau  de  reussite  des  operations  d’influence,  en  particulier  dans  les  milieux  hostiles 
ou  il  pourrait  s’averer  difficile  et  peu  concluant  de  realiser  des  entrevues  et  des  sondages  d’opinion  par 
des  methodes  traditionnelles.  Cependant,  il  a  ete  affirme  que  les  effets  decisifs  des  operations  d’influence 
etaient  observables  et,  par  le  fait  meme,  mesurables.  Le  present  rapport  etudie  les  defis,  les  limites  et  les 
possibilites  de  revaluation  des  effets  et  du  niveau  de  reussite  des  operations  d’influence  effectuees  aupres 
de  populations  civiles  en  milieux  hostiles.  11  traite  egalement  des  limites  des  methodes  utilisees 
actuellement,  ainsi  que  des  possibilites  liees  aux  mesures  discretes.  La  demarche  se  fonde  sur  un  examen 
systematique  de  documents  issus  des  domaines  de  la  psychologie,  de  l’anthropologie  et  de  la  sociologie, 
ainsi  que  sur  des  entrevues  menees  aupres  de  pionniers  des  technologies  de  l’intelligence  artificielle 
utilisees  par  exemple  pour  les  systemes  d’alerte  rapide  lors  de  conflits.  11  serait  possible  de  recourir  a 
l’intelligence  artificielle  des  medias  sociaux  afin  d’evaluer  Tefficacite  et  le  niveau  de  reussite  des 
operations  d’influence  en  milieux  hostiles,  en  particulier  lorsque  les  attitudes  ou  les  comportements  sont 
delicats  sur  le  plan  social  et  ne  se  pretent  pas  bien  a  T auto-evaluation  des  participants,  ou  lorsque  les 
capacites  de  collecte  des  donnees  avant  la  tenue  des  operations  sont  limitees  et  qu’il  est  impossible 
d’etablir  une  base  de  reference. 
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